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(57) ABSTRACT 

Embedding a watermark in an image by changing selected 
DCT coefficients in the blocks and macro blocks of coeffi- 
cients which represent the image. The changes in the blocks 
that comprise each macro block are done in a coordinated 
manner so that the phase of the watermark signal is pre- 
served across the block boundaries. By preserving the phase 
across block boundaries, a detectable grid is formed which 
can be used as an orientation and scaling grid. Furthermore, 
by preserving the phase across block boundaries the visual 
artifacts introduced by the watermark are minimized. The bit 
rate of the image signal is preserved by maintaining a count 
(referred to as the cumulative change count) that represents 
the amount that the bit rate has been increased by changes 
in coefficients less the amount that the bit rate has been 
decreased by changes in the coefficients. If at any time the 
cumulative change count exceeds a pre-established limit, 
coefficient changes that decrease the cumulative change 
count continue; however, coefficient changes that increase 
the cumulative change count are suspended. The suspension 
of coefficient changes that increase the cumulative change 
count continues until the cumulative change count falls 
below the pre-established limit. 

2 Claims, 4 Drawing Sheets 
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METHOD AND APPARATUS FOR 
WATERMARKING VIDEO IMAGES 

RELATED APPLICATIONS 
This application is: 

1) a continuation in part of Patent application No. 60/056, 
968, which was, filed on Aug. 8. 1997 and which is now 
pending, 

2) a continuation in part of application Ser. No. 08/746, 
613 filed Nov. 12, 1996 which is now U.S. Pat. No. 
6,122,403 and 

3) a continuation in part of application Ser. No. 08/649, 
419 which was filed May 16, 1998 which is now U.S. 
Pat. No. 5,862,260. 

FIELD OF THE INVENTION 

This invention relates to stenography, to copy protection 
and to applying and detecting digital watermarks in video 
images. 

BACKGROUND OF THE INVENTION 

The advent of technology for storing images on digital 
media has increased the need for a method to protect against 
piracy. Images stored on prior forms of media (e.g. VHS, 
Beta, audiotapes, etc) are inherently degraded when copied. 
Images stored on digitally encoded media can be copied 
with no degradation; therefore, perfect copies of copies of 
copies, etc. can be made. 

The introduction of Digital Versatile Discs (DVD) con- 
taining movies has created increased incentives for both 
casual and professional unauthorized copying. At the movie 
industry's urging, technology has been put in place to 
protect against simple duplication of DVD disks using 
equipment available to unsophisticated consumers. This is 
similar to the protection that exists which prevents one from 
duplicating a VCR tape by connecting together two com- 
mercially available VCRs. 

While such protection mechanisms protect against some 
types of copying, a personal computer connected to a DVD 
device present a much more complicated problem. Open 
architecture devices such as personal computers reproduce 
the signals in the "clear" and such devices have many entry 
points, which can be used to duplicate material once it is in 
the "clear". The present invention uses digital watermarks to 
address the above described problem. The present invention 
also has other applications. 

It is known that to facilitate the detection of digital 
watermarks one can insert a watermark signal that forms a 
grid. The grid can be used to determine orientation and scale. 
With the present invention the data signal and the grid signal 
are integrated into a single watermark signal in such a 
manner that the visual artifacts introduced by the watermark 
are minimized. 

In applications such as DVD, an important factor that 
needs be considered is the bit rate of the bit stream. There are 
disadvantages if introduction of a watermark into a bit 
stream changes the bit rate. For example if images are going 
to be recorded on a medium such as a DVD disc, increasing 
the number of bits in the bit stream will decrease the number 
of images that can be recorded on a single disk. It is known 
that, in general, adding a watermark to a stream of images 
will increase the number of bits in the bit stream. The present 
invention provides a method and apparatus, which preserves 
the bit rate even though watermarks are introduced into the 
images. 
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SUMMARY OF THE INVENTION 

The well-known JPEG and MPEG data compression 
techniques transform images utilizing a direct cosine trans- 

5 form (DCT) which produces a matrix of DCT coefficients. 
These coefficients are arranged into blocks (e.g. into 8 by 8 
blocks of coefficients). The blocks of DCT coefficients are in 
turn arranged into macro blocks (e.g. into 16 by 16 arrays 
containing four 8 by 8 blocks). With the present invention 

lQ selected DCT coefficients in each block are slightly 
increased or slightly decreased in response to a watermark 
signal. The changes in the blocks that comprise each macro 
block are done in a coordinated manner so that the phase of 
the watermark signal is preserved across the block bound- 

35 aries. By preserving the phase across block boundaries, a 
detectable grid is formed which can be used as an orientation 
and scaling grid. 

The present invention also maintains the bit rate of the 
image signal. The bit rate of the signal is preserved by 

20 maintaining a count (referred to as the cumulative change 
count) that represents the amount that the bit rate has been 
increased by changes in coefficients less the amount that the 
bit rate has been decreased by changes in the coefficients. If 
at any time the cumulative change count exceeds a pre- 

2 5 established limit, coefficient changes that decrease the 
cumulative change count continue; however, coefficient 
changes that increase the cumulative change count are 
suspended. The suspension of coefficient changes that 
increase the cumulative change count continues until the 

30 cumulative change count falls below the pre-established 
limit. The above described process can be described as 
selectively changing the intensity of a watermark signal in a 
bit stream so as to prevent the entropy of the combined 
signal from exceeding a pre-established limit. 

35 BRIEF DESCRIPTION OF THE FIGURES 

FIG. 1 is a diagram illustrating how the pixels in an image 
are arranged into blocks and how the resulting DCT coef- 
ficients are numbered. 
40 FIG. 2 is a diagram, which shows how the blocks of DCT 
coefficients are arranged into macro blocks. 

FIG. 3 is a program flow diagram showing how the 
coefficients in each block of a macro block are treated so as 
to preserve the phase of watermark signal in each macro 
block. 

FIG. 4 is a program flow diagram showing how the bit rate 
in the data stream is maintained constant. 

DESCRIPTION OF PREFERRED 
50 EMBODIMENTS 

The well know MPEG (Motion Picture Expert Group) and 
JPEG (Joint Photographic Expert Group) image compres- 
sion techniques use a DCT (Discreet Cosine Transform) to 

55 generate a matrix of coefficients. The preferred embodiment 
of the invention shown herein slightly modifies the DCT 
coefficients (either slightly increases or slightly decreases 
the value of the coefficients) so as to embed a digital 
watermark in the image. Such a digital watermark can later 

60 be detected by conventional cross correlation techniques. 
As illustrated in FIG. 1, the MPEG and JPEG techniques 
divide an image into 8 by 8 blocks of pixels. Each block of 
pixels is then used to generate an eight by eight block of 
DCT coefficients. The 8 by 8 blocks of coefficients are 

65 divided into "macro blocks", each of which consist of four 
of the original blocks. This is illustrated in FIG. 2. The rows 
and columns of DCT coefficients in each block are num- 
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bered from top to bottom and left to right as illustrated in 
FIG. 1. The first row and the first column are designated as 
the "0" row and "0" column. 

Certain of the DCT coefficients in each block are selected 
as the coefficients that will carry a selected bit of the digital 
watermark signal. In the preferred embodiment the three 
coefficients circled in FIG. 1 are used to carry the first or "0" 
bit of the watermark data signal. These three coefficients are 
modified, that is, either slightly increased or slightly 
decreased depending upon the value of the "0" bit of the 
watermark data. In a similar manner other coefficients are 
slightly changed in order to carry the other bits of the 
watermark signal. 

One aspect of the present invention is directed to insuring 
that the sinusoids generated by the changes made to the DCT 
coefficients are continuous, that is, in-phase across the four 
blocks that constitute each macro block. First, if the sinu- 
soids that carry the watermark are continuous across each 
macro block, there will be less edge effects and the water- 
mark will be less visually noticeable. Second, the sinusoids 
which are continuous over the four blocks of each macro 
block create a low level orientation or grid signal. This low 
level grid signal can be detected to determine the orientation 
and scale of the watermark. The grid signal can then be 
detected using the cross correlation techniques. Cross cor- 
relation detection techniques are for example shown in 
copending patent application 08/649,149 filed May 16, 1996 
and in issued U.S. Pat. Nos. 5,748,763 and 5,748,782. 

If certain DCT coefficients in adjacent blocks are modified 
in the same direction, the resulting sinusoids will not be 
continuous across block boundaries. With the present inven- 
tion the changes made to the coefficients of each of the four 
blocks in a macro block are coordinated so that the resulting 
sinusoids will be continuous across block boundaries within 
each macro block. The changes are coordinated using the 
rules explained below. 

The blocks in each macro block are numbered as shown 
in FIG. 2. Block one is considered the base block. The 
coefficients in this block are changed in a conventional way 
by the associated bits of the watermark signal. Note, the 
following paragraphs relate to how the coefficients which 
are circled in FIG. 1 are changed in response to the "0" bit 
of the watermark. It should be understood that other coef- 
ficients must be similarly changed to carry the other bits in 
the watermark data. 

In block X 7 the coefficients that are circled in FIG. 1 are 
slightly increased or slightly decreased in response to the 
"0" bit of the watermark data. In blocks 2, 3 and 4, the 
circled coefficients shown in FIG. 1 are changed in response 
to the zero bit of the watermark according to the following 
rules. 

Block 2: invert the direction of the change if the coeffi- 
cient is in an odd row. 
Block 3: invert the direction of the change if the coeffi- 
cient is in an odd column 
Block 4: invert the direction of the change if the coeffi- 
cient is in an odd row or if it is in and odd column, but 
do not invert the direction of the change if the coeffi- 
cient is in both and odd row and in an odd column. 
If the above rules are followed the sinusoids generated by 
the change in the DCT coefficients will be continuous across 
the boundaries in the four blocks that form each macro 
block. These sinusoids will be able to be detected using 
conventional cross correlation techniques and they can be 
used as a grid to determine the scale and rotation of the 
image. The data bits in the watermark will also be able to be 
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detected using conventional watermark detection tech- 
niques. Thus, the watermark data itself is used to form the 
grid that can be used to determine scale and rotation. 

FIG. 3 is an overall program flow diagram of the above 
described aspect of the preferred embodiment The system 
accepts a stream of data that represents images. Block 301 
is a conventional device or program module that generates 
DCT coefficients for the images in the data stream. These 
10 coefficients are sent to a decision unit 302 which separates 
the data into macro blocks and sends it to units 311, 312, 313 
and 314 depending upon whether the data represents a block 
1, 2, 3 or 4 in a macro block. Units 311-313 modify the DCT 
coefficients in order to imbed a watermark signal according 
15 to the following rules. 

Unit 311: modify the coefficients in a conventional man- 
ner to imbed watermark. 
Unit 312: invert the direction of the change if the coef- 
20 ficient is in an odd row. 

Unit 313: invert the direction of the change if the coef- 
ficient is in an odd column 
Unit 314: invert the direction of the change id the coef- 
25 ficient is in an odd row or if it is in and odd column, but 
do not invert the direction of the change if the coeffi- 
cient is in both and odd row and in an odd column. 
The output of units 322 to 314 is combined by unit 320 
back into a single data stream. It is noted that each of the 
30 units shown in FIG. 3 could be separate units, which are 
either programmed, or hardwired to perform the specified 
functions. Alternatively all the function could be performed 
in a single programmed computer on a time-shared basis. 
35 The unit which generates DCT coefficients is conventional 
and such units are know and not part of the present inven- 
tion. 

The previous discussion describes how a watermark can 
be introduced in the DCT domain. It is noted that the 

40 durability of the overall watermarking can be increased by 
using two watermarks. One watermark can be added by 
modification of the pixels in the original image in the 
manner as described in U.S. Pat. Nos. 5,748,763 or 5,748, 

45 783 and then a second watermark can be added by modifi- 
cation of the coefficients in the DCT domain as described 
herein. 

Another problem addressed by the present invention is the 
need to maintain a constant bit rate in a stream of bits 

50 representing a series of images even though watermarks are 
added to the images. It is noted that MPEG and JPEG 
systems use variable length codes to represent data, hence, 
adding watermarks generally increases the bit rate of a data 
stream. Typical a watermark has no correlation with the 

55 image into which the watermark is embedded, thus embed- 
ding an image in a watermark produces an image which has 
a higher entropy than the original image. The bit rate of a 
data stream transmitting an image correlates directly to the 

60 entropy of the image. 

Typically the number of codes used to code an image, that 
is, the number of entries in the Huffman table of a coded 
image, is relatively large (e.g. 500). However, the changes 
that occur when a watermark is introduced into an image can 

65 be illustrated with following simple example. Consider a 
data stream that has only four symbols, si, s2, s3 and s4, 
which are encoded as follows: 
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Symbol code 


si 


0 


S2 


01 


S3 


110 


S4 


111 



Then consider a data stream as follows: 

Bit stream: 0011010111010 

Decoded stream 0/0/110/10/111/0/10 

Decoded message: si, s2, s3, s2, s4, si, s2 

When a watermark is added to an image the bits in the 
image are slightly changed. In the above simplistic illustra- 
tive example, in some situation the symbol s2 would be 
changed to the symbol s3 and hence the number of bits in a 
bit stream which transmits the image would be increased. In 
fact there are mathematical principles (not explained herein) 
which show that when a normally distributed watermark 
(that is, a watermark with a Gaussian distribution) is added 
to an image, and the image is transmitted using variable 
length Huffman codes, the length of the bit stream will of 
necessity be increased. 

The present invention provides a technique for insuring 
that when a watermark is added to a data stream, the bit rate 
will be maintained constant. It is noted that the present 
invention does not violate the above-described mathematical 
principle, because with the present invention, some of the 
redundancy normally used to watermark images is in certain 
circumstances decreased. That is in certain circumstances 
the intensity of the watermark is decreased. 

With the present invention, the watermark is modified in 
response to characteristics of the image. Thus, to some 
extent the watermark is correlated to the image into which 
the watermark in embedded. In this way a watermark can be 
embedded into an image and the entropy of the combined 
image and watermark will be substantially equal to the 
entropy of the watermark alone. 

With the present invention, the system maintains a cumu- 
lative count of the amount that the coefficients have been 
changed to any point in time. That is, the amount of positive 
changes less the amount of negative changes made since the 
beginning of the bit stream is tracked. This amount is herein 
referred to as the cumulative change count. If at any time, 
the cumulative change count exceeds a pre-established posi- 
tive limit, no further positive changes are made. 

Normally it is only necessary to insure that changes do not 
increase the bit rate unduly; however, in some instances it 
may also be desirable to insure that changes do not unduly 
■decrease the bit rate. If this is desired, the same technique as 
described above can be used to insure that the cumulative 
change amount does not exceed a pre established negative 
limit. That is, if the cumulative change amount exceeds a 
pre-established negative value, positive changes continue in 
a normal manner, but no further negative changes are made. 

The magnitude of the pre-established maximum (and in 
both a positive and negative direction) are established at the 
values which constitutes the change in bit rate which can be 
tolerated in a particular system. 

FIG. 4 is a program flow diagram showing how the data 
rate is maintained constant not withstanding the fact that 
watermarks are added to the images in the data stream. 
Block 403 A shows that a limit on the amount of positive 
changes that can be made to DCT coefficients is established 
and stored. Blocks C shows that the cumulative change 
amount is stored. The cumulative change amount is the 
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amount of positive changes less the amount of negative 
changes that have been made to coefficients since the start of 
the data stream. 

The DCT coefficients are calculated in the normal manner 

5 as indicated by block 401. Likewise the change in each the 
DCT coefficients needed to embed the watermark is also 
calculated in the normal manner as shown by block 405. 
Block 405 shows that a check is made to determine if the 
needed change in a particular DCT coefficient is positive or 

10 negative. Block 409 indicates that if the change is positive 
a check is made to determine if the maximum allowable 
cumulative change amount stored in block 403A will be 
exceeded if the change is made. 

Blocks 415, and 417 indicate that the coefficients will 

15 only be changed, if the change does not cause the cumulative 
change amount to exceed the limit in 403A. Finally as 
indicated by blocks 425 and 427, the cumulative change 
amount in register 403C is incremented or decremented if a 
change to the coefficients is in fact made. Block 431 

20 indicates that the coefficients are sent to the output of this 
process and they are then transmitted and processed in a 
normal manner. 

It is noted that the present invention relates to embedding 
a watermark in an image. Various know techniques can be 

25 used to detect watermarks embedded in images utilizing the 
present invention. For example techniques such as those 
described in U.S. Pat. Nos. 5,748,763, and 5,748,783 or in 
the "Communications of the ACM" July 1998/vol. 41, No .7. 
or in pending applications Ser. No. 08/746,613 filed Nov. 12, 

30 1996 and Ser. No. 08,649,419 which was filed May 16, 1998 
(all of which are hereby incorporated herein by reference) 
could be used. 

While the process has been described above as one where 
a change is either made or not made, it should be understood 

35 that alternatively, the amount of the change could be 
decreased if the limit in the cumulative change value is being 
approached. It is also noted that the system shown in FIG. 
4 prevents the cumulative change value from exceeding a 
pre established positive limit. Since adding a watermark to 

40 an image generally increases the entropy of the image and 
since Huffman code tables are normally constructed such 
that an increase in entropy result in increased bit rate, the use 
of only a positive limit is normally appropriate. However, in 
some situations, it may be appropriate to tract if the cumu- 

45 lative change amount exceeds a limit in both the positive and 
negative directions. Such a check could be added to FIG. 4 
prior to block 427. 

It is recognized that by implementing the present 
invention, the strength of the watermark is in some cases 

50 reduced. However, the reduction is not sufficient to prevent 
detection of the watermark. The changes made with the 
above invention merely lower the intensity of the watermark 
in a selective manner, thus in some instances more process- 
ing may be required to detect the watermark. 

55 In many systems, each Huffman code covers several 
symbols. In such systems the calculation indicated by block 
405 is not the change in a single symbol that results from 
adding a watermark to the image. In such systems the 
calculation indicated by block 405 is a calculation of the 

60 change that results in the bit string of whatever combination 
of symbols used in the Huffman code to represent a symbol. 
In some cases the calculation might have to be done over 
several combinations of symbols. 

It is also noted that various aspects of the present inven- 

65 tion are shown herein in a single preferred embodiment. 
Other alternative embodiments could use one but not all 
aspects of the present invention. For example the part of the 
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present invention that relates to maintaining bit rate could be 
used in embodiments which do not use macro blocks to 
establish an orientation grid. Likewise the aspect of the 
present invention which relates to the use of macro blocks 
could be used without the part of the invention that relates 5 
to maintaining a constant bit rate. Finally, while the inven- 
tion has been shown in an embodiment that inserts a 
watermark in the DCT domain, the invention could be used 
in applications where watermarks are inserted in other 
domains. ]0 

While the invention has been shown and described with 
respect to preferred embodiments of the invention, various 
changes in form and detail could be made without departing 
from the spirit and scope of the invention. The applicants 
invention is limited only by the appended claims. 
We claim: 15 
1. A method for adding a mul libit watermark to an image 
comprising, 

generating DCT (Discrete Cosine Transform) coefficients 
representing said image, said coefficients being 
arranged in blocks, said blocks being arranged into 20 
macro blocks, 

each of said macro blocks containing a block one, a block 
two, a block three and a block four, and the coefficients 
in each block being arranged in rows and columns, 
alternate of said rows and columns being designated 25 
even and odd rows and columns, adjusting said coef- 
ficients as follows: 

the coefficient in block one being changed directly in 
response to the bits of said watermark, the bits in blocks 
two, three and four being changed in accordance with 30 
the following rules: 

Block two: invert the direction of the change if the 
coefficient is in an odd row, 

Block three: invert the direction of the change if the 
coefficient is in an odd row column 35 

Block four: invert the direction of the change if the 
coefficient is in an odd row or if it is in and odd 
column, but do no invert the direction of the change 
if the coefficient is in both and odd row and in an odd 
column, 
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whereby the sinusoids generated by said adjustments are 
in phase across block boundaries in each macro block. 
2. A system for adding a multibit watermark to an image 
comprising, 

means for generating DCT (Discreet Cosine Transform) 
coefficients representing said image, said coefficients 
being arranged in blocks, said blocks being arranged 
into macro blocks, 
each of said macro blocks contains four blocks, block one, 
block two, block three and block four, the coefficients 
in said blocks being arranged in rows and columns, 
alternate of said rows and columns being designated 
even and odd rows and columns, 
means for adjusting said coefficients in accordance with 
the bits of said watermark to embed said watermark in 
said image, 

the coefficient in block 'one being changed directly in 
response to the bits of said watermark, the bits in blocks 
two, three and four being changed in accordance with 
the following rules: 

Block two: invert the direction of the change if the 

coefficient is in an odd row, 
Block three: invert the direction of the change if the 

coefficient is in an odd column 
Block four: invert the direction of the change if the 
coefficient is in an odd row or if it is in and odd 
column, but do not invert the direction of the change 
if the coefficient is in both and odd row and in an odd 
column 

whereby the sinusoids generated by said adjustments are 
in phase across block boundaries in each macro block, 
and 

whereby the same changes which represent said water- 
mark form a grid which can be used to detect the scale 
and rotation of said watermark. 

***** 
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